Eye Beautification Under Inaccurate Localization

ABSTRACT

Sub-regions within one or more face images are identified within a digital image, and enhanced by applying an artificial glint symmetrically and/or synchronously to image data corresponding to sub-regions of eyes within the face image. An enhanced face image is generated including an enhanced version of the face that includes certain original pixels in combination with pixels corresponding to the one or more eye regions of the face with the artificial glint.

PRIORITY and RELATED APPLICATIONS

This application claims benefit under 35 U.S.C. §120 as a Continuationof application Ser. No. 13/969,558, filed on Aug. 17, 2013, which is acontinuation-in-part (CIP) of U.S. patent application Ser. No.12/827,868, filed Jun. 30, 2010, which is a continuation in part (CIP)of U.S. patent application Ser. No. 12/512,796, filed Jul. 30, 2009,which claims the benefit of priority to U.S. provisional patentapplication No. 60/084,942, filed Jul. 30, 2008. This application isalso related to U.S. Ser. No. 12/512,819 and U.S. Ser. No. 12/512,843.The entire contents of each of which are hereby incorporated byreference for all purposes as if fully set forth herein.

BACKGROUND

1. Field of Invention

The invention relates to image processing, particularly of detected eyeregions within face images.

2. Description of the Related Art

Proctor and Gamble's U.S. Pat. No. 6,571,003 mentions finding and fixingfacial defects such as spots, wrinkles, pores, and texture insub-regions of faces, e.g, cheeks or in areas defined by landmark pointssuch as corner or nose, eye, or mouth. The technique involves replacingthe defined region with a mask. The P&G patent discloses toelectronically alter the color.

The P&G patent also mentions detecting and correcting lighting gradientsand lighting variances. These lighting gradients, or variances, appearto involve instances where there is directional lighting which may causea sheen or brighter region on the facial skin. U.S. patent applicationSer. Nos. 12/038,147, 61/106,910 and 61/221,425, which are assigned tothe same assignee as the present application and are hereby incorporatedby reference, describe techniques which use Viola-Jones type classifiercascades to detect directional lighting. However, determining andcorrecting a lighting gradient would typically involve global analysis,exceptions being possible in combination with face-tracking techniquessuch as those described at U.S. Pat. Nos. 7,403,643 and 7,315,631 andU.S. application Ser. No. 11/766,674, published as 2008/0037840, andSer. Nos. 12/063,089, 61/091,700, 61/120,289, and 12/479,593, which areall assigned to the same assignee as the present application and arehereby incorporated by reference. It is desired to have a technique thatuses a local blurring kernel rather than such techniques involving lessefficient global analysis for certain applications and/or under certainconditions, environments or constraints.

Kodak's U.S. Pat. No. 7,212,657 illustrates at FIGS. 13-14 to generate ashadow/peak image (based on generating a luminance image and an averageluminance image), a blur image, and blended images. The Kodak '657patent states that a shadow/highlight strength image is generated bysubtracting an average luminance image from a luminance image. Also, atFIG. 16, the Kodak '657 patent shows element 1530 is labeled as“generate luminance and chrominance scaling factors using peak/valleymap and color info”, and element 1540 is labeled as “modify luminanceand chrominance of pixels within mask regions”. Face detection isdescribed in the Kodak patent, but not face tracking.

The Kodak technique, like the P&G technique, involves global imagemanipulations, i.e., the “luminance image” is not indicated as includinganything less than the entire image, the “blur image” involves theapplication of a kernel to the entire image, and the “blended image”involves three copies of the global image. The “blur image” involveschrominance and luminance data meaning that a lot of memory is used formanipulating the image, particularly if the application involves aresource constrained embedded system. Regarding luminance andchrominance scaling factors, even if they involve localized scalingfactors, they are not described in the Kodak patent as being generatedfor application to anything less than the entire image.

U.S. patent application Ser. Nos. 11/856,721 and 12/330,719, which areassigned to the same assignee as the present application and are herebyincorporated by reference, describes a technique that can be applied asa single, raster-like, scan across relevant regions of an image withoutinvolving global analysis or a determination of global properties suchas the average luminance image, or a shadow or blur image. Suchsingle-pass scan through predetermined regions provides a far moreefficient and suitable technique for embedded systems such as digitalcameras than either of the P&G or Kodak patents.

The Hewlett Packard (HP) published patent application 2002/0081003mentions air-brushing which typically involves applying color over aswath of an image, e.g., such as may include a blemish or wrinkle. TheHP publication also mentions blurring over a wrinkle on an image of aperson's face, and again specifically describes blurring or blendingcolor values defining the wrinkles and surrounding skin. The HPapplication mentions changing brightness to brighten or darken a facialfeature, such as to shade a facial feature, and goes on to describechanging color values of skin associated with the feature to shade thefeature. The HP patent further discloses to sharpen a hair line and/orblur a forehead and/or cheeks, by blurring color values. Face detectionand face tracking over multiple images, full resolution or lowresolution and/or subsample reference images such as previews, postviewsand/or reference images captured with a separate imaging system before,during or after capturing of a main full-resolution image are notdescribed in the HP patent, nor is there any suggestions to smooth orblur luminance data of a digital face image.

Portrait is one of the most popular scenes in digital photography. Imageretouching on portrait images is a desirable component of an imageprocessing system. Users can spend a lot of time with conventionalsoftware trying to make a portrait nicer by hiding wrinkles andblemishes. It is desired to provide an innovative automatic portraitscene enhancer, which is suitable for an embedded device, such as adigital still camera, camera-phone, or other handheld or otherwiseportable consumer appliance having image acquisition components (e.g.,lens, image sensor) and a processor.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor.

Copies of this patent or patent application publication with colordrawing(s) will be provided by the Office upon request and payment ofthe necessary fee.

FIG. 1A-1B illustrate unprocessed and processed images of a face, wherethe processing involves applying selective smoothing or blur to regionsof the face.

FIGS. 2A-2C illustrate identification of regions of a face, andprocessed and unprocessed version of a face image, wherein theprocessing involves application of selective smoothing or blurring ofcertain regions of the face.

FIGS. 3A-3B illustrate low-resolution skin maps that may be used in aface beautification procedure, including a raw skin-map at FIG. 3A and aresult after smoothing and thresholding at FIG. 3B.

FIGS. 4A-4B illustrate an eye bounding box and projection of intensityinformation (Y coordinate in the YUV space) onto the horizontal axis,wherein rough eye limits, a center of the eye, and a rough eye whitecoordinate are each approximately determined.

FIGS. 5A-5B illustrate an intensity profile along a horizontal line(FIG. 5B) through the eye illustrated at FIG. 5A.

FIGS. 6A-6B illustrate two masks used for upwards (left) and downwards(right) path growing, where C=current pixel and I=inspected pixel.

FIG. 7 illustrates an example of iris border found by a procedure inaccordance with certain embodiments.

FIG. 8 illustrates the three lines used in an eye white determinationprocess in accordance with certain embodiments.

FIG. 9 illustrates an intensity profile determined using the three linesof FIG. 8 in an eye white determination.

FIG. 10 illustrates a determined silhouette of an eye white inaccordance with certain embodiments.

FIGS. 11A-11D illustrate eye beautification process in accordance withcertain embodiments.

FIGS. 12A-12D illustrate another eye beautification process inaccordance with certain embodiments.

FIG. 13A illustrates an original facial image or a previously corrected(e.g., flash-induced eye defect corrected) facial image.

FIG. 13B illustrates the image of FIG. 13A with artificial soft boxglints added to the pupils of the eyes in the facial image in accordancewith certain embodiments.

FIG. 13C illustrates the image of FIG. 13A with artificial on-cameraflash glints added to the pupils of the eyes in the facial image inaccordance with certain embodiments.

FIG. 13D illustrates the image of FIG. 13A with artificial umbrellaglints added to the pupils of the eyes in the facial image in accordancewith certain embodiments.

FIG. 13E illustrates the image of FIG. 13A with artificial beauty dishand reflector glints added to the pupils of the eyes in the facial imagein accordance with certain embodiments.

FIG. 13F illustrates the image of FIG. 13A with artificial ring flashglints added to the pupils of the eyes in the facial image in accordancewith certain embodiments.

FIGS. 14A-14B illustrate examples of added artificial glints at reportedeye centers with asymmetric eye center localization.

FIGS. 15A-15B illustrate examples of added artificial glints at reportedeye centers with synchronized or symmetric eye center localization inaccordance with certain embodiments.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Techniques are provided in accordance with several embodiments describedherein that enhance an appearance of a face within a digital image. Oneor more groups of pixels are identified that each include a pupil regionof an eye region within the face in the digital image. A border betweena pupil region and an iris or a sclera is identified using luminanceinformation (and/or optionally chrominance information, and/or sizeand/or shape information). The digital image is adjusted by adding oneor more glint pixels at a pupil side of the border between the iris andthe pupil to generate an enhanced image, which is itself, or a furtherprocessed version is, displayed, transmitted, communicated, rendered ordigitally stored or otherwise output.

The face within the digital image may be detected using luminanceinformation. Skin tone portions of the face may be segmented from facefeatures including eye regions. Pixels of the iris may be enhanced at aniris side of the border between the iris and the pupil. One or morelocalized color smoothing kernels may be applied to the iris, such thatthe iris of the enhanced image may include pixels modified from originalpixels of the face at least by localized color smoothing. Noisereduction or enhancement, or both, may be applied to the iris, such thatthe iris of the enhanced image comprises pixels modified from originalpixels of the face at least by localized noise reduction or enhancement,or both. A border point may be found as a maximum gradient point along ahorizontal line through an upright eye, and the border may be grownincluding adding immediate neighboring points characterized by largesthorizontal intensity gradients. The border between the pupil and theiris or the sclera of the eye may be found using color information.

One or more non-transitory processor readable media are also providedthat has or have code embedded therein for programming one or moreprocessors to enhance an appearance of a face within a digital image ofa scene including a face that has been captured using a lens and animage sensor of a same or different device, wherein the code isconfigured to program the processor to perform any of the methodsdescribed herein. A digital image acquisition device is also providedthat includes a lens and image sensor for capturing digital images, aprocessor, and one or more such media.

Eye beautification is provided herein as an advantageous step ofportrait beautification, as providing a subject a more youthful andhealthy look. An automatic eye beautification technique is providedhereinbelow, which complements the automatic face beautificationprocedure set forth at US2010/0126831 which is assigned to the sameassignee and is hereby incorporated by reference. In the following, anautomatic eye beautification procedure is presented that is both fast asinvolving low computational complexity, and effective, even as producingsatisfactory results for both Caucasian and Asian races, and even forimages of low quality (acquired with what might be considered by some tobe low-end cameras or small camera component add-ons to other deviceslike phones, music/video players, portable computers and the like.Embodiments are described which are particularly useful with embeddedapplications. Eye beautification is provided that works for Asiansubjects when conventional techniques often do not. In certainembodiments, eye locations are detected inside a detected face area.Borders are detected between an iris and a sclera. An outer border ofthe sclera is detected. The sclera and/or iris are beautified.

Embodiments are described to automatically enhance an aspect of eyes indigital images including portrait pictures. In accordance with certainembodiments, a method includes detecting an eye white and an eye iris,and then improving their aspect and/or one or more furthercharacteristics such as enhancing color, luminance, overall and/orrelative exposure, tone reproduction, white point, color balance, blur,focus, orientation, size, and/or contrast, or one or more othercharacteristics, and/or removal of blemishes or other defects such asblinking, cross-eyes, looking away, flash-induced red eye, golden eye,white eye and/or zombie eye, iris shape or size, red eye of the scleraand/or one or more other defects.

A technique is provided for enhancing an appearance of a face within adigital image using a processor. A digital image of a scene is acquiredincluding a face. The image is captured using a lens and an image sensorof a processing device, and/or the image is received following captureby another device that includes a lens and an image sensor. The face isdetected within the digital image. Skin tone portions of the face aresegmented from face features including one or two eyes. Within at leastone of the eyes, a border is identified between an iris and a sclera ofthe eye using luminance information. Pixels of the iris are enhanced atan iris side of the border. The enhanced image or a further processedversion is displayed, transmitted, communicated or digitally stored orotherwise output.

The enhancing of the iris may include linear contrast stretching and/orspatial blurring.

The technique may further include applying one or more localized colorsmoothing kernels to the iris, such that the iris of the enhanced imagecomprises pixels modified from original pixels of the face at least bylocalized color smoothing.

The technique may further include applying noise reduction orenhancement, or both, to the iris, such that the iris of the enhancedimage comprises pixels modified from original pixels of the face atleast by localized noise reduction or enhancement, or both.

The technique may further include finding a border point as a maximumgradient point along a horizontal line through an upright eye, andgrowing the border including adding immediate neighboring pointscharacterized by largest horizontal intensity gradients.

One or more further segments of a periphery of the sclera may beidentified. The technique may further include enhancing pixels of thesclera within the periphery. The identifying of one or more furthersegments of a periphery of the sclera may include determining pointsalong multiple horizontal lines, wherein the points are located each ata next local intensity minimum that follows a local intensity maximum.

With or without identifying the periphery of the sclera, the techniquemay involve enhancing pixels of the sclera at a sclera side of theborder.

The identifying the border between the iris and the sclera of the eyemay be performed without using color information.

Another technique is provided for enhancing an appearance of a facewithin a digital image using a processor. A digital image is acquired ofa scene including a face. The image is captured using a lens and animage sensor of a processing device, and/or the image is receivedfollowing capture by another device that includes a lens and an imagesensor. The technique involves detecting a face within the digitalimage. Skin tone portions of the face are segmented from face featuresincluding one or two eyes. Within at least one of the eyes, a border isidentified between an iris and a sclera of the eye using luminanceinformation. Pixels of the sclera are enhanced at a sclera side of theborder. The method further includes displaying, transmitting,communicating or digitally storing or otherwise outputting the enhancedimage or a further processed version, or combinations thereof.

The technique may include applying one or more localized color smoothingkernels to the sclera, such that the sclera of the enhanced imagecomprises pixels modified from original pixels of the face at least bylocalized color smoothing.

The technique may include applying noise reduction or enhancement, orboth, to the sclera, such that the sclera of the enhanced imagecomprises pixels modified from original pixels of the face at least bylocalized noise reduction or enhancement, or both.

The technique may include identifying one or more further segments of aperiphery of the sclera. The enhancing of pixels of the sclera mayinvolve utilizing information regarding said periphery. The identifyingof one or more further segments of a periphery of the sclera may involvedetermining points along multiple horizontal lines, wherein the pointsare located each at a next local intensity minimum that follows a localintensity maximum.

The identifying of the border between the iris and the sclera of the eyemay be performed without using color information.

A digital image acquisition device is also provided, including a lens,an image sensor and a processor, and a processor-readable memory havingembodied therein processor-readable code for programming the processorto perform any of the methods described herein.

One or more computer-readable media are provided that have embodiedtherein code for programming one or more processors to perform any ofthe methods described herein.

Using at least one reference image, and in certain embodiments more thanone reference image, including a face region, the face region may bedetected. In those embodiments wherein multiple reference images areused, a face region may be tracked. Face detection and tracking areperformed preferably in accordance with one or more techniques describedin the US patents and US patent applications listed above and below andwhich are incorporated by reference here.

Given an input image and one or more, or two or more, smaller,subsampled, and/or reduced resolution versions of the input image (e.g.,one QVGA and one XGA), the position of a face and of the eyes of theface within the input image may be determined using face detection andpreferably face tracking. FIG. 1A shows an example of an unprocessedimage including a face, or at least an image of a face with originalimage data or image data that has been otherwise processed than byselective localized smoothing or blurring such as is described withreference to embodiments herein. The face beautification method mayinvolve applying a selective blur, and/or other smoothing such aslocalized averaging or according to one or more of the methodsspecifically described below, which enhances the skin, e.g., softeningand/or reducing wrinkles and/or spots. FIG. 1A illustrates anunprocessed image of a face before applying selective smoothing. FIG. 1Billustrates a processed version of the image of the face of FIG. 1A,i.e., after applying selective smoothing to certain sub-regions of theface.

In an exemplary embodiment, the method may be performed as follows.Certain sub-regions of the face are identified, e.g., rectangularsub-regions or other polygonal or curved or partially-curved sub-regionswith or without one or more cusps or otherwise abrupt segmentalintersections or discontinuities. These sub-regions may be places whereit will be desired to apply selective smoothing, or these sub-regionsmay be those places outside of which it is desired to apply theselective smoothing, or a combination of these. For example, threesub-regions such as two eyes and a mouth may be identified for notapplying selective smoothing, and/or four sub-regions such as aforehead, two cheeks and a chin may be specifically selected forapplying localized luminance smoothing.

Now, in the embodiment where the two eyes and mouth are identified, theskin around these facial sub-regions/rectangles may be detected. Thiscan include in certain embodiments creating a binary skin image,including segmenting the QVGA version of the image. In one embodiment,this involves thresholding done in YCbCr.

A larger rectangle or other shape may be defined around the face as awhole. That is, outside of this larger facial shape, it may be desiredin most embodiments herein not to apply the selective smoothing(although there may be other reasons to smooth or blur a background orother region around a detected face in a digital image, such as to blura background region in order to highlight a face in the foreground; see,e.g., U.S. Pat. No. 7,469,071 and US2009/0040342, which are assigned tothe same assignee and are hereby incorporated by reference). A skin mapmay be filtered by morphological operations. The largest regions insidethe face may be selected to be kept, and regions may be selected basedon other criteria such as overall luminance, a certain thresholdluminance contrast such as may be indicative of wrinkled skin, a colorqualification such as a certain amount of red, a spotty texture, oranother unsatisfactory characteristic of a region or sub-region of aface. Lip detection may be performed based on color information (Crcomponent) and/or on the position of the eyes, nose and/or ears or otherface feature such as chin, cheeks, nose, facial hair, hair on top ofhead, or neck, and/or on a shape detector designed for specificallydetecting lips.

The skin inside of one or more face regions, not including the eye andmouth regions, may be corrected. In certain embodiments this involvesskin pixels from inside a face region having their luminance componentreplaced with different luminance values, such as an average value ofits neighbors, e.g., substantially all or a fair sampling of surroundingskin pixels, or all of a majority of pixels from one direction as if thepixels were being replaced by blurred pixels caused by relativecamera-object movement in a certain direction. Smoothing can include anaveraging process of skin pixels from other regions of the face, and/orcan be a calculation other than averaging such as to prioritize certainpixels over others. The prioritized pixels may be closest to the pixelbeing replaced or may have a color and/or luminance with greatercorrelation to a preferred skin tone.

Certain criteria may be applied as requirement(s) for correcting aregion within an image. For example, it may be set as requisite that theregion be inside a face, although alternatively the skin of a person'sneck, leg, arm, chest or other region may be corrected. It may be set asrequisite that the luminance component be within a certain range. Thatrange may depend on an average luminance of the skin within the certainface or a preferred luminance or a selected luminance. The certain pixelmay be selected or not selected depending on its relation with otherdetails within the face (e.g., eyes, nose, lips, ears, hair, etc.). Thenumber of neighbors used when modifying the current pixel (i.e., thekernel size) may be varied depending on the size of the face versus thesize of the image, or on a standard deviation of luminance values,and/or other factors may be taken into account such as the resolution ora determination as to how much fixing the particular face region orsub-region ought to receive. If the face is too small compared to theimage (e.g., the face uses below a threshold percentage of the availablepixel area, then the system can be set to apply no correction ofwrinkles, spots, etc., because such undesired features may not bevisible anyway. The averaging or other smoothing or blurring may be doneon a XGA image in order to improve speed.

Localized Blurring/Smoothing Kernel(S)

The blurring kernel or smoothing kernel in certain embodiments may bechanged, adjusted, selected, and/or configured based on one or morefactors specific to the image and/or group of images based upon which acorrected image is to be generated. A factor may be relative size of thefacial image to that of the main picture. Other factors may includeresolution of the face region and/or the entire image, processingcapacity and/or RAM or ROM capacity, and/or display, projection ortransmission capacity of an embedded device or processing or renderingenvironment with which the image is acquired, processed and/or output.

The blurring kernel may include a table, formula, calculation and/orplot of face sizes (e.g., 5% of image, 10% of image, 20% of image, etc)versus kernel sizes (e.g., 3×3, 4×4, 5×5, etc.) The kernel may also beadjusted based the relative location of the sub-region within a face.The kernel applied to the cheeks may be configured to blur cheekseffectively, while a different kernel to apply to the skin around theeyes may be configured to blur/smooth that skin most effectively, samefor the skin in the forehead, the skin around the mouth/chin, etc. Adifferent kernel can be applied to a bearded region or other hair regionor no smoothing may be applied to such regions. In a specific, simpleexample embodiment, the blurring/smoothing kernel is smaller when facesare smaller (two or more levels or one or more thresholds may be used).The blurring kernel may decrease working around eyes or lips or nose orbearded regions or low luminance regions or dark colored regions. Theblurring kernel may depend on average luminance around the point ofinterest.

The method in certain embodiments may include the application ofselective skin enhancement and/or noise removal. This provides analternative approach to determining the facial regions when abeautification filter or blurring/smoothing kernel might not be applied.

An Alternative Implementation Lee-Based Filtering

A face beautifier may use certain relevant data gathered in a facetracking technique as described in reference cited herein andincorporated by reference (see below). That information may include aposition of the face and/or a feature within the face such as one orboth eyes, mouth or nose, information relating to where skin is detectedand its tone, luminance, shaded areas, direction relative to incominglight, etc. That data can also include the Cb, Cr, Y range within theface area, and/or backlighting image information.

Application to Luminance Channel

The technique according to certain embodiments may employ modificationsof the luminance channel to achieve the filtering of the skin. Datarelating to variance within the luminance channel may also be used, andtexture information of the skin of the face region or sub-region may beused. Such texture information may include certain chrominance data, butmay also include only luminance data which defines such texture withinthe image. The variance on luminance may be utilized when selectingand/or performing blurring/smoothing, and may be applied specifically toseparating wrinkles (which are typically rather isolated) from thetexture of the face of a shaved man or even an unshaved man (wherevariance is high). The texture information may involve a measure of towhat degree areas or sub-regions are uniform or not. The textureinformation may include a recognized or learned or newly-analyzedpattern, which can be analyzed either on the luminance channel onlyand/or also on one or more color channels.

In certain embodiments, only face and eyes may be mandatory, while inothers certain other features may be required. Face tracking may be usedbut is not required for the technique to provide tremendous advantage inbeautifying a face. The location of a face within an image may begathered using face detection only or using face tracking. A dynamicskin-map and/or contrast info may be gathered using face tracking.

Within a digital camera or real-time imaging appliance, a real-time facetracking subsystem (operable on a sequence of preview, postview or otherreference images independent of the main image) may be operated, and onacquisition of a main image, facial enhancements may be performed basedon (i) an analysis of the facial region in the main acquired image and(ii) an analysis of face region metadata determined from the real-timeface tracking subsystem.

Facial Image Enhancement

Apart from the image to be enhanced, the algorithm may use (ifavailable) extra information, including the position of the face(s) andeyes in the given image which will help limiting the area of search, andtwo resized copies of the initial image (e.g.: one QVGA and one XGA).These two images may be used for faster processing power where accuracyis less critical.

An example algorithm according to certain embodiments may be describedas follows:

Enhancement Map Detection

Based on face information, skin tones similar to those inside a facerectangle are sought in the entire image. In detail, for each facepassed, the steps may be as follows in one example embodiment (notnecessarily in the order discussed below):

Compute the average saturation for the region of interest (entire facerectangle or other shape in this case). To avoid problems in cases suchas side illumination, the average saturation for the entire image mayalso be computed and the minimum between the two may be used.

The relevant skin tone information (from the face rectangle) isextracted. This is done by geometrical considerations (and additionallyby color filtering). In one implementation this means: top, left andright of the rectangle are changed in such a way that 1/5 of each sideis not taken into account. Bottom (based on image orientation) stays thesame or not depending on whether it is deemed important to have the neckincluded. One implementation of color filtering may be the eliminationor reduction of luminance or change of color of pixels which aredetermined to have non-skin tones (e.g. blue pixels).

PCA (Principal Component Analysis) procedure may be applied on remainingpixels. A pixel may be given by a triplet. The covariance matrix of thegiven pixels is computed. The eigenvectors and eigenvalues of thecovariance matrix are then found. The three resulting eigenvectorsrepresent the axes of a new 3D coordinate system. The two leastimportant axes (corresponding to the two smallest eigenvalues) arefurther considered.

The coordinates of all inspected pixels on the two abovementioned axesare computed. The two histograms of the absolute value of thecoordinates are then computed: one histogram for each axis. For each ofthe two histograms, an acceptance threshold may be determined, forexample, using the following procedure. The corresponding cumulativehistogram H is computed. The threshold is taken such as to delimit agiven percentage of the total number of pixels (i.e., threshold Th istaken such as H(Th)˜=p %, with p being a predefined value). By choosingdifferent values for p one can vary the strength of the skin filtering.For example values taken for p may vary from 90.0% (for strongfiltering) up to 97.5% (for permissive filtering).

Compute the coordinates of each image pixel on the two axes resultingafter the PCA step and check if the absolute values are smaller than thethresholds obtained in the previous step.

For a pixel to be considered skin type further verification may be done.An example is to check that saturation is large enough in the YUV colorspace. Based on the average saturation computed in the first stage, eachpixel may be verified to have at least one of the U and V values largeenough. Also the luminance level of the pixel is checked to be in apredefined gamut. This is because we do not want to beautify dark hairor too bright areas where color information is not reliable.

In the same time, a generic skin detection algorithm (e.g. simplethresholding on the YUV space) may be applied on the entire image toobtain a less reliable but more inclusive skin map. The role of thegeneric skin map may be manifold, as it may replace the PCA skin map incases where face information is not present. The skin map may also beused to improve the PCA skin map by helping in deciding if holes in themap are going to be filled. The skin map may add up to the PCA skin map“uncertain skin pixels”, or pixels with a lower confidence which are tobe treated separately by the correction block.

The skin map may now be cleaned up by applying spatial filtering such asmorphological operations. At this point the skin map may have two levelsof confidence: PCA skin (high confidence) and uncertain skin (lowconfidence). The number of levels of confidence may be further increasedby taking into consideration the spatial positioning of a skin pixelinside the skin area. In one implementation, the closer one pixel is tothe interior of the map, the higher its confidence is set. In anotherimplementation, the number of skin confidence levels could be increasedfrom the PCA thresholding stage by using multiple thresholding of pixelcoefficients on the PCA axes.

Enhancement Map Correction

The skin pixels from inside the faces (or the ones from regions thatpassed skin filtering when no face is present) may be corrected inaccordance with certain embodiments. An example process for performingthis correction is described below.

A weight αε[0,1]α may be computed for each pixel describing how muchcorrection it will receive. The higher the value of α, the morecorrection will be applied to that pixel. The weight may be based on thelocal standard-deviation computed on the XGA intensity image over asquared neighborhood (e.g. 16×16 for large-size skin areas, or 8×8 formedium-sized skin areas), but may also take into account other factors(e.g., the skin level of confidence, the proximity of the pixel to facefeatures, such as eyes and mouth etc.)

Initially, α is computed as:

${= \frac{\sigma_{skin}}{\sigma_{local}}},$

where σ_(skin) is the standard deviation computed over the whole skinarea, while σ_(local) is the local standard deviation. Then α is limitedto 1.

α may be increased by a predefined factor (e.g., 1.1-1.25) for pixelshaving higher confidence of skin.

α may be decreased by a predefined factor for pixels located in thevicinity of face features, such as eyes and mouth (see FIG. 1). (For eyeand mouth detection, see chapter on eye and mouth beautification).

Special attention may be given to pixels located near the skin border.In this example, for those pixels, σ_(local) is higher owing to the factthat there is a strong edge in the computing neighborhood. In thesecases, the direction of the edge is sought (only the four maindirections are considered) and, based on it, the most uniform sub-windowof the current window is used for recomputing a and the local average.

The parameter a may also modified based on the relationship between theintensity of the current pixel and the local average (computed over thesame neighborhood as σ_(local)). This is because face artifacts that areattempted to be eliminated by face beautification (e.g, freckles,pimples, wrinkles) may be typically darker than skin, but not very dark.

In one embodiment, the following modification may be performed: if thecurrent intensity is greater than the local average, decrease a (highintensity, therefore, strongly reduce correction). If the currentintensity is much lower than the local average, decrease a (too dark tobe a face artifact, strongly reduce correction). If the currentintensity is lower than the local average, but the difference betweenthe two is small, increase α (very likely face artifact, thereforeincrease correction). If the current intensity is lower than the localaverage, and the difference between them is important, slightly decreaseα (less likely to be a face artifact, therefore slightly reducecorrection).

Apply correction on the intensity value, based on the relation:

NewIntensity=α·LocalAverage+(1−α)·OldIntensity

The averaging may be computed on the same intensity image used for theweighting map (XGA image). This improves speed without affectingquality.

FIGS. 2A-2C illustrates an example of working with detected features. InFIG. 2A, input and predetermined data are illustrated with colorsincluding cyan (blue-ish hue) for the face rectangle, green for facefeatures such as eye and mouth or lips, and red for skin inside the facearea.

FIG. 2B illustrates an initial image, and FIG. 2C illustrates an outputresult using auto face beautification.

Enhancement of Facial Features Eyes and Mouth

Besides removing skin artifacts (such as wrinkles, pimples etc.), eyesand mouth beautification may be applied as well towards an overallbetter visual aspect of the face. The following actions may be taken foreye and mouth beautification.

Initial locations of eyes and mouth may be (coarsely) determined as thelargest holes in the PCA skin map located in the upper left, upper rightand lower half parts of the face rectangle or other shape.

More precise eye and mouth localization may be performed at a higherresolution (XGA at least) in a small neighborhood surrounding theinitial areas described above, as follows:

Mouth Beautification

A mouth area may be detected based on color information. When using YUVcolor space, it may be defined as the area which has the V componenthigher than a threshold (computed based on the local V histogram).

The presence of teeth may be checked by inspecting the histogram ofsaturation S inside the smallest rectangle surrounding the mouth area.If working in YUV color space, saturation may be computed asS=abs(U)+abs(V). If the histogram of saturation is unimodal, then teethmight not be visible. If the histogram of saturations is bimodal, thenthe area corresponding to the inferior mode of the histogram may beinspected. If this area is found to be located inside the mouth area(more precisely, if a sandwich mouth-teeth-mouth is present), then itmay be decided that teeth are visible.

The mouth redness may be increased. In YUV color space this may be doneby multiplying the V value inside the mouth area by a predefined factor(e.g., 1.2).

The teeth may be whitened by slightly increasing the Y component whilereducing the absolute value of U and V components.

Eye Beautification

One or both eye areas may be detected each as a connected area that hasthe normalized Y·S component lower than a threshold (computed based onthe local Y·S histogram). In the above expression, Y is the normalizedintensity component from the YUV color space, whereas S is thenormalized saturation, computed as above. Normalization of both Y and Smay be done with respect to the local maximum values.

The iris may be detected as the central, darker part of the eye, whereassclera (eye white) may be detected as the remaining part of the eye.

The eye white may be brightened and whitened, by slightly increasing theY component while reducing the absolute value of U and V componentsinside the eye white area. Further examples of eye beautification areprovided in detail below.

The iris may be improved by stretching the intensity contrast inside theiris area. Also, if the red eye phenomenon is present (which results inan increased V value of the pupil area located inside the iris), a redeye correction algorithm may be applied, as may a golden eye algorithm(see, e.g., U.S. Pat. Nos. 6,407,777, 7,042,505, 7,474,341, 7,436,998,7,352,394, 7,336,821 and 7,536,036, which are incorporated byreference).

In accordance with several embodiments, the quality of portrait imagesmay be improved by doing face, skin and/or face feature enhancement.

Rough Determination of the Eye Position

The starting data is the low resolution (typically a 240×320 thumbnail)skin map used for the face beautification procedure. A smoothed skin-mapis computed by smoothing the skin map with a 5×5 averaging kernel andthresholding. FIGS. 3A-3B illustrate a low-resolution skin map used in aface beautification procedure in accordance with certain embodiments.FIG. 3A illustrates a raw skin-map, while FIG. 3B illustrates aresultant map that is obtained after smoothing and thresholding. Then,the rough eye location may be identified as the largest holes in theskin map located in the upper left, and upper right corners of the faceregion. A bounding box (e.g., which may be enlarged by a few linesand/or columns in an effort to ensure that it includes the whole eye)may be considered for further processing in the aim of preciselydetecting the eye boundary. In certain embodiments, full resolutionimage data in the bounding box may be considered, while in others, asubsampled version may be used.

Rough Determination of Horizontal Coordinates of Eye Center and Limits

The following operations are applied in certain embodiments on the Ycomponent inside the bounding box computed in a rough determination ofeye position. A projection of the eye may be computed onto a horizontalaxis (optionally, it may be blurred, e.g., with a 1×5 averaging kernel).An example of the form of such a projection may be as presented in FIG.4A. Based on this illustrative projection, five pints of interest may becomputed in accordance with certain embodiments. A coordinate of theminimum value MV (middle/black dot in FIG. 4B) is determined as anapproximate value of the eye center. Two coordinates WC (white-scleracenters) of relative maxima are determined that are situated at the leftand right of the minimum MV as approximations of the centers of the twoparts of the eye white. Two coordinates EB (eye boundaries) of relativeminima situated at the left and right of green points (in red) as theapproximation of the horizontal limits of the eye. FIGS. 4A-4Billustrate an eye bounding box and projection of the intensityinformation (Y coordinate in the YUV space) onto the horizontal axis.Rough eye limits EB, a center of the eye MV, and rough eye white centerscoordinates WC are each approximately determined.

Iris Border Determination

Consider 4-5 horizontal lines equally spaced from one another, whosevertical position is located near the center of an image of an eye. Oneexample of such a line HL is illustrated in FIG. 5A. FIG. 5B illustratesan intensity profile along the horizontal line HL through the eyeillustrated at FIG. 5A. Candidate iris borders are approximated. Asearch area is restricted in the interval delimited by the approximationillustrated at FIG. 4B.

For each line, margins IM of the iris are determined on that line as thepoints having the largest negative (on the left) and positive (on theright) horizontal gradient situated inside the interval between thewhite-sclera center points WC approximated at FIG. 4B. Only the pointsIM having the absolute value of the gradient maximum among the 4-5 lines(one at the left and one at the right) are retained. Those two points IMwill be starting point for the iris border finding procedure explainedbelow.

Starting from the two selected iris border points IM, the whole irisborders may be determined by the following approach: from each of thetwo points, two paths (upwards and downwards) may be grown by addingimmediate neighboring points characterized by the largest horizontalgradient. For the upward path, only the three upper pixels will beinspected in the embodiment illustrated at FIG. 6A, where C=currentpixel and I=inspected pixel), and for the downward path illustrated atFIG. 6B, the three corresponding lower pixels will be inspected. Thepath growing procedure may be ended when the absolute value of thegradient is smaller than one quarter of the gradient in the initialpoint. An example of an iris border IB found by the above-describedprocedure is presented in FIG. 7.

Determination of Eye White

For most close-up portraits of Caucasian people taken with camerashaving a decent quality, the eye white can be separated from skin mainlyby color and intensity. Yet, this is not the case for Asian people, forwhich typical eye white is both yellowish (thus, easily causing falsepositives with a skin detector) and dark, given the small opening of theeye that characterizes the Asian race. Given this, the eye whitedetector described below does not take color into account, and is basedinstead on intensity. In this manner, overall results are better, andresults on Caucasian (statistically) can be enhanced in certainembodiments by taking color into account.

The steps of eye white finding are illustrated at FIG. 8. Three pointsare determined on each of the two borders of the iris, including amiddle point, and two points located at ¼ and ¾ of the border,respectively. Each pair of corresponding points (on the left and rightiris border) are used to draw one of the three lines TL illustrated atFIG. 8.

For each line, the borders of the eye white are determined by analyzingan intensity profile along the line. Borders of the eye white aredetermined as points located at the next local minimum that follows alocal maximum (the same scenario may be applied to both the left andright side of the line) on a path emerging from the corresponding borderpoint. An example in shown in FIG. 9, where the intensity profile alongthe middle line of the three lines TL of FIG. 8 is presented. FIG. 9includes two border points BP, two local maxima, LMa, and two localmimima, LMi, which are taken as eye white borders.

For each side of the iris (left and right), with the two ends of theiris border (high, low) and the three eye white borders determined asper the procedure above, the silhouette of the eye white is drawnbetween the points. An example eye white-sclera border EWB isillustrated at FIG. 10.

Beautification

The eye white may be whitened (e.g., Y increased by a factor of 1.1-1.2,U and V decreased by a factor of 3-4). Best results may be obtained ifthe factors are not spatially constant, and instead decrease towards theouter border of the eye white. The iris pixels (after optionallyremoving glint and pupil, i.e., too dark and too bright pixels) may thenundergo a linear contrast stretching procedure. After applyingcorrection, a spatial blurring procedure may be applied at the inner andouter border of each of the modified regions (i.e., iris and eyewhite-sclera).

Two examples of correction in accordance with these embodiments areillustrated at FIGS. 11A-11D. FIG. 11A illustrates an original eyeimage, while FIG. 11B illustrates a corresponding intensity imageshowing the borders of the iris and the three lines described above.FIG. 11C illustrates a mask of eye white and iris, while a final,enhanced eye is illustrated at FIG. 11D. FIGS. 12A-12D illustrateanother example (note that a red eye reduction procedure has not beenapplied in this example, but would be optionally applied in certainembodiments).

Alternative Embodiments

Certain embodiments benefit very advantageously when provided on digitalcamera and especially on a handheld camera-equipped device. Usingspecific data from a face detector, or even a face tracker (with datafrom multiple image frames) can permit the method to performadvantageously. In one embodiment, an enhanced face image may beacquired dynamically from a face tracker module. The use of a PCA todetermine main skin color can be advantageous, as well as using the twoother color space dimensions to determine variation from that color. Themethod may include decorrelating the color space into “primary skin” and“secondary skin”. The use of the “secondary skin” dimensions todetermine “good skin” can be advantageous for skin detection as well. Asmaller image may be used for the detection, while the localizedsmoothing kernel(s) may be applied to the full image, thereby savingvaluable processing resources to great advantage on a handheld device.Two skin maps may be used, including an “exclusive” one combined with an“inclusive” one, and face detection data may also be utilized. Many“skin analysis” and tone/color/contrast and other image adjustmenttechniques may be combined with embodiments described herein, e.g. asdescribed at US published application no. 2006/0204110, which isincorporated by reference. Skin and facial feature detection (eyes,mouth) is advantageously used in facial image enhancement, which mayinclude smoothing, blur, texture modification, noisereduction/enhancement, or other technique for reducing a visual effectof a blemish or blemished region of a face. Wrinkle correction may beeffected within certain embodiments.

In addition, PCA-based “strong” skin detection may be advantageouslyutilized, which enables detection of only those skin tones which aresimilar to those of the face, and may be used to discard other skin-likepatches whose color is yet different from that of the skin (e.g., a wallbehind, light hair, etc.).

The embodiments described herein utilize application of selectivesmoothing which is not to all skin pixels of the face, but only to thosewhich are likely to be or include artifacts (e.g., wrinkles, pimples,freckles etc.). This is very different from global solutions where allfacial skin pixels or the entire face are smoothed and facial non-skinpixels (e.g. mouth, eyes, eyebrows) are sharpened. These embodimentsserve to preserve intrinsic skin textures, while removing unwantedartifacts. For instance, a person's will look their age, thus remainingnatural, while still improving the appearance of the face.

Age Classification

In another embodiment, a processor-based digital image acquisitiondevice is provided, e.g., with a lens and image sensor, a processor andcode for programming the processor to perform a method of enhancingacquisition parameters of a digital image as part of an image captureprocess using face detection within said captured image to achieve oneor more desired image acquisition parameters. Multiple groups of pixelsthat correspond to a face within a digitally-acquired reference imageare identified. Values are determined of one or more attributes of theface. One or more default image attribute values are compared with oneor more of the determined values. The face is classified according toits age based on the comparing of the image attribute values. A cameraacquisition parameter is adjusted based on the classifying of the faceaccording to its age. A main image is captured in accordance with theadjusting of the camera acquisition parameter.

The process may also include generating in-camera, capturing orotherwise obtaining in-camera a collection of low resolution imagesincluding the face, and tracking said face within said collection of lowresolution images. The identifying of face pixels may be automaticallyperformed by an image processing apparatus. Automated processing of theface pixels may be performed based on the classifying.

The camera acquisition parameter may include exposure. The age of theface may be classified as that of a child, baby, youth, adult, elderlyperson, and/or may be determined based on recognition of a particularface. The adjusting of the camera acquisition parameter may includereducing exposure. Fill-flash may be applied to the face inpost-processing. The adjusting of camera acquisition parameter mayinclude optimizing focus on a baby's or child's or youth's face,centering the face, increasing the size of the face, cropping around theface, adjusting the orientation or color of the face, or combinationsthereof, and/or may involve increasing the resolution and/or reducingthe compression of pixels of the face of the baby or child or otherclassification of face.

The face may be tracked over a sequence of images.

A method is provided for enhancing an appearance of a face within adigital image using a processor. An image is acquired of a sceneincluding a face. The face is identified within the digital image. Oneor more sub-regions to be enhanced with localized luminance smoothingare identified within the face. One or more localized luminancesmoothing kernels are applied each to one of the one or more sub-regionsidentified within the face to produce one or more enhanced sub-regionsof the face. The one or more localized smoothing kernels are applied toluminance data of the one or more sub-regions identified within theface. An enhanced image is generated including an enhanced version ofthe face including certain original pixels in combination with pixelscorresponding to the one or more enhanced sub-regions of the face. Theenhanced image and/or a further processed version is displayed,transmitted, communicated and/or digitally stored and/or otherwiseoutput.

The localized luminance smoothing may include blurring or averagingluminance data, or a combination thereof.

One or more localized color smoothing kernels may be applied to the oneor more sub-regions. The one or more enhanced sub-regions of thecorrected image may also include pixels modified from original orotherwise processed pixels of the face at least by localized colorsmoothing.

Noise reduction and/or enhancement may be applied to the one or moresub-regions. The one or more enhanced sub-regions of the corrected imagemay also include pixels modified from original or otherwise processedpixels of the face at least by localized noise reduction and/orenhancement.

Certain non-skin tone pixels within the one or more sub-regions of theface may be determined not to have a threshold skin tone. These non-skintone pixels may be removed, replaced, reduced in intensity, and/ormodified in color.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which comprise one or more functions of arelationship between original pixel intensities and local averageintensities within the one or more original and/or enhanced sub-regions.

One or more mouth and/or eye regions may be detected within the face. Anatural color of one or more sub-regions within the one or more mouthand/or eye regions may be identified and enhanced. These sub-regions mayinclude one or more teeth, lips, tongues, eye whites, eye brows, irises,eye lashes, and/or pupils.

The face may be classified according to its age based on comparing oneor more default image attribute values with one or more determinedvalues. One or more camera acquisition and/or post-processing parametersmay be adjusted based on the classifying of the face according to itsage.

A digital image acquisition device is also provided, including a lens,an image sensor and a processor, and a processor-readable memory havingembodied therein processor-readable code for programming the processorto perform any of the methods described herein, particularly forenhancing an appearance of a face or other feature within a digitalimage.

One or more processor-readable media are also provided that haveembodied therein code for programming one or more processors to performany of the methods described herein.

In certain embodiments, face tracking using previews, postviews or otherreference images, taken with a same or separate imaging system as a mainfull resolution image is combined with face beautification. Thisinvolves smoothing and/or blurring of face features or face regions,wrinkle/blemish removal, or other digital cosmetic adjustments. Incertain embodiments, a luminance channel is used for smoothing anunsightly feature, while in a narrow subset of these, only the luminancechannel is used for smoothing without using any color channel. Otherembodiments used one or more color channels in addition to the luminancechannel, and these may or may not also use face tracking.

In certain embodiments, localized modification of a region of a face isperformed based on an average of the pixel values surrounding aparticular pixel. This localized averaging/blurring kernel may beapplied solely on the luminance channel, thereby reducing computation inan embedded system such as a portable digital camera, camera-phone,camera-equipped handheld computing device, etc.

A single-pass filtering kernel may be configured to act only on localluminance values within pre-determined regions of the image, and may becombined with a binary skin map. This is far more efficient, using lessmemory and executing more quickly, within an embedded imaging systemsuch as a digital camera.

Blurring or shading may be achieved by changing selected luminancevalues of one or more sub-regions of a face. An embodiment involvesapplying or subtracting luminance over a swath of an image, e.g., suchas may include a blemish or wrinkle. Blurring may also be applied to afacial feature region that includes a wrinkle on an image of a person'sface. Blurring and/or blending luminance values of a face featureregion, e.g., temple region, side of nose, forehead, chin, cheek region)defining the wrinkles and surrounding skin. Brightness may be changed tobrighten or darken a facial feature, such as to shade a facial feature,and this may be achieved by changing luminance values of skin associatedwith the feature to shade or brighten the feature.

In certain embodiment, a technique is provided including in-cameraprocessing of a still image including one or more faces as part of anacquisition process. The technique includes identifying a group ofpixels including a face within a digitally-acquired still image on aportable camera. One or more first processing portions of the image isdetermined including the group of pixels (the first portion may becharacterized as foreground). One or more second processing portions ofthe image other than the group of pixels is then determined (and may becharacterized as background). The technique may include automaticallyin-camera processing the first processing portion with a determinedlevel of smoothing, blurring, noise reduction or enhancement, or otherskin enhancement technique involving one or more luminance components ofthe pixels, while applying substantially less or no smoothing, blurring,noise reduction or enhancement or otherwise to the second processingportion to generate a processed image including the face. The processedimage or a further processed version including the face is then stored,displayed, transmitted, communicated, projected or otherwise controlledor output such as to a printer, display other computing device, or otherdigital rendering device for viewing the in-camera processed image. Themethod may include generating in-camera, capturing or otherwiseobtaining in-camera a collection of low resolution images including theface, and determining the first processing portion including analyzingthe collection of low resolution images. The analyzing may includetracking the face within the collection of low resolution images.

A further method is provided for enhancing an appearance of a facewithin a digital image. A digital image of a scene including a face isacquired using a processor. The image is captured using a lens and animage sensor, and/or the image is received following capture by a devicethat includes a lens and an image sensor. The face is identified withinthe digital image. Skin tone portions of the face are segmented fromface features including one or two eyes or a mouth or combinationsthereof. Within the skin tone portions of the face, one or more blemishregions that vary in luminance at least a threshold amount fromnon-blemished skin tone portions are identified. Luminance data of theone or more blemish regions is smoothed to generate smoothed luminancedata. An enhanced image is generated including an enhanced version ofthe face that has original luminance data of the one or more blemishregions replaced with the smoothed luminance data and combined withoriginal non-blemished skin tone portions. The enhanced image and/or afurther processed version is/are displayed, transmitted, communicated,digitally stored and/or otherwise output.

The localized luminance smoothing may include blurring and/or averagingluminance data.

The method may include applying one or more localized color smoothingkernels to the one or more sub-regions. The one or more enhancedsub-regions of the corrected image further may include pixels modifiedfrom original pixels of the face at least by localized color smoothing.

The method may include applying noise reduction or enhancement, or both,to the one or more sub-regions. The one or more enhanced sub-regions ofthe corrected image may include pixels modified from original pixels ofthe face at least by localized noise reduction and/or enhancement.

The method may include determining certain non-skin tone pixels withinthe one or more sub-regions that do not comprise a threshold skin tone,and removing, replacing, reducing an intensity of, or modifying a colorof said certain non-skin tone pixels, or combinations thereof.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which comprise one or more functions of arelationship between original pixel intensities and local averageintensities within the one or more original and/or enhanced sub-regions.

The method may include detecting one or more mouth and/or eye regionswithin the face, and identifying and enhancing a natural color of one ormore sub-regions within the one or more mouth or eye regions, includingone or more teeth, lips, tongues, eye whites, eye brows, iris's, eyelashes, or pupils, or combinations thereof.

A further method is provided for enhancing an appearance of a facewithin a digital image. A processor is used to generate in-camera,capture or otherwise obtain in-camera a collection of one or morerelatively low resolution images including a face. The face isidentified within the one or more relatively low resolution images. Skintone portions of the face are segmented from face features including oneor two eyes or a mouth or combinations thereof. Within the skin toneportions of the face, one or more blemish regions are identified thatvary in luminance at least a threshold amount from the skin toneportions. A main image is acquired that has a higher resolution than theone or more relatively low resolution images. The main image is capturedusing a lens and an image sensor, or received following capture by adevice that includes a lens and an image sensor, or a combinationthereof. The method further includes smoothing certain original data ofone or more regions of the main image that correspond to the same one ormore blemish regions identified in the relatively low resolution imagesto generate smoothed data for those one or more regions of the mainimage. An enhanced version of the main image includes an enhancedversion of the face and has the certain original data of the one or moreregions corresponding to one or more blemish regions replaced with thesmoothed data. The enhanced image and/or a further processed versionis/are displayed, transmitted, communicated and/or digitally stored orotherwise output.

The method may include tracking the face within a collection ofrelatively low resolution images.

The smoothing may include applying one or more localized luminancesmoothing kernels each to one of the one or more sub-regions identifiedwithin the face to produce one or more enhanced sub-regions of the face.The one or more localized luminance smoothing kernels may be applied toluminance data of the one or more sub-regions identified within saidface. The localized luminance smoothing may include blurring and/oraveraging luminance data. The method may also include applying one ormore localized color smoothing kernels to the one or more sub-regions.The one or more enhanced sub-regions of the corrected image may includepixels modified from original pixels of the face at least by localizedcolor smoothing.

The method may also include applying noise reduction and/or enhancementto the one or more sub-regions. The one or more enhanced sub-regions ofthe corrected image may include pixels modified from original pixels ofthe face at least by localized noise reduction and/or enhancement.

Certain non-skin tone pixels may be determined within the one or moresub-regions that do not comprise a threshold skin tone. The method mayinclude removing, replacing, reducing an intensity of, and/or modifyinga color of such non-skin tone pixels.

Enhanced pixels of the one or more enhanced sub-regions may includeenhanced intensities which have one or more functions of a relationshipbetween original pixel intensities and local average intensities withinthe one or more original and/or enhanced sub-regions.

One or more mouth and/or eye regions may be detected within the face. Anatural color may be identified and enhanced for one or more sub-regionswithin the one or more mouth and/or eye regions, including one or moreteeth, lips, tongues, eye whites, eye brows, iris's, eye lashes, and/orpupils.

Eye Beautification Under Inaccurate Localization

In accordance with certain embodiments, an artificial glint, or specularreflection artifact, is added to one or more eye regions, e.g., in thecenter of each of the eyes, in a portrait image or an image thatincludes a face as a significant feature therein. In certainembodiments, a natural glint region may have been overwhelmed by aflash-induced artifact such as red-eye, golden eye, white, eye, zombieeye or the like. In other cases, the glint region may have been“painted-over” during a process of image enhancement and/or image defectcorrection.

In one approach, a glint region is searched for at each of the centersof eyes in a digital image. White pixels or high intensity pixels intheir luminance data are searched for that are surrounded by dark pixelsor black pixels or pupil and/or iris and/or sclera region pixels inluminance, color and/or shape. If the glint region is not found, then incertain embodiments the glint region is added by in-painting at or nearthe center of the eye region.

In other embodiments, in cases wherein it may be known that a pupilregion does not have an original glint region, e.g., because an imageenhancement and/or defect correction process has been performed on oneor more eye regions within a digital image, then a glint region may beselected and inserted into the digital image, e.g., at or near thecenter of the eye region. In certain embodiments, eye centerlocalization may be performed and an artificial glint may be added inthe center of the eye. An in-painting technique may be used, and/orcolors of certain black pixels may be changed to white, and/orluminances of certain pixel shapes may be increased.

An example of a digital image including eye regions that do not haveglint regions is shown in FIG. 13A. Examples of artificial glintconfigurations that have been added to the eye regions in FIG. 13A areprovided in FIGS. 13B-13F. In these examples, the correction strengthmay be tuned for highest pleasing effect. FIG. 13B illustrates a “softboxes” artificial glint configuration. FIG. 13C illustrates an on-cameraflash artificial glint configuration. FIG. 13D illustrates an“umbrellas” artificial glint configuration. FIG. 13E illustrates a“beauty dish and reflector” artificial glint configuration. FIG. 13Fillustrates a ring flash artificial glint configuration.

In certain embodiments, added glint regions are configured as lightsthat are consistent with 3D scene physics. Accordingly, the artificialglint regions are configured in these embodiments symmetrically for botheyes with respect to iris center and/or pupil center and/or centers ofeye regions determined by using one or more shapes of eye lids, scleras,eye brows, partial face or full face regions and/or horizontal and/orvertical gaze direction determinations (in this context, U.S. Pat. No.8,212,864 and US published patent applications 2013/0057553,2013/0057573, 2012/0075483, 2012/0219180 and 2012/0218398 areincorporated by reference). Moreover, in certain embodiments, one ormore glint regions are added to one or more eye centers consistent witha scene illumination point, i.e., not to appear to come unnaturally fromthe shadows.

FIGS. 14A-14B illustrate examples of artificial glints added in thereported eye centers. However, in FIG. 14A, clearly the right eye of thesubject has a glint added that is too low and both added glints may betoo large in diameter to appear natural. Moreover, in FIG. 14B thehorizontal gaze is not centered but is somewhat to the right of thesubject. This off-center gaze, or perhaps the method of finding the eyecenter, has or have caused the artificial glint to appear in the irisregion rather than in the pupil region for the left eye of this subject.In both cases, the asymmetry of localization may be considered to bevisually disturbing. Moreover, the correction strength may be good forvisibility, but can appear unnatural when it is too intense.

Eye center localization methods may also suffer from imperfections whenthere are shadows, occlusions, eye glasses, and/or expressions that maybe asymmetric (in this context, Hansen,” In the Eye of the Beholder: ASurvey of Models for Eyes and Gaze”, IEEE Trans on PAMI, is incorporatedby reference). A lack of precision may be corroborated by a key requestthat can cause the solution of placing the glint in the determined eyecenters to be prone to errors.

Therefore, in certain embodiments, determination of the eye centers isnot relied upon. Instead, points that are alike in the eye region aredetermined. Eye center localization may be performed as a startingpoint, but then a matching of the positioning within similar eye regionsmay be performed, e.g., to localize pixels in the centers of the regionsthat are similar enough. This has the advantageous effect of providing asymmetry for glint regions that appear in the two eyes of a subject in adigital image, which is a natural-looking result.

In certain embodiments, a face detection technique is applied to thedigital image, or information regarding positions and/or sizes and/orshapes and/or colors of eye regions that have already been identified,for example in red eye detection or iris detection processes, may beused. An eye center localization and/or radius identification techniquemay be applied to identified eye regions. In this way, the centers ofboth the left and the right eyes of a subject are available andsynchronized together so that symmetry can be achieved in the glintinsertion process. An example may be as follows:

A common radius value may be determined or defined as a mean of the twoiris radiuses. A center of the left (or right) eye may be set as areference pixel. It may be reasonable to presume that sometimes it willbe shifted from the true eye center. A region may be set around thisreference eye center pixel (e.g., pixRow±common radius; pixCol±commonradius). A found center may be determined now for the right (or left)eye, i.e., the other of the two eyes. Then, a neighborhood centered onthe right (or left) eye center having a same size, shape, luminance,and/or color as the left eye region is determined. Now, if the tworegions are deemed to be similar enough or a check is performed todetermine that the two regions are similar enough, then a selected glintshape, size, luminance and/or color may be added to both eye regions,i.e., in symmetric fashion.

In certain embodiments, a search around the reference pixel of one orboth eyes may be used to determine the location of whose neighborhoodprovides a better similarity or a maximum similarity. The similaritybetween two image patches may be measured or otherwise determined inaccordance with one of the many similarity measures, such as sum ofabsolute differences, sum of square differences, cross-correlationand/or “SSIM” such as may be described in Z. Wang, A. C. Bovik, H. R.Sheikh and E. P. Simoncelli, “Image quality assessment: From errorvisibility to structural similarity,” IEEE Trans. on Image Processing,which is incorporated by reference. A value for SSIM larger than 0.2 maybe selected to provide sufficient similarity.

FIGS. 15A-15B illustrate examples when the artificial glint is notplaced in the centers of the eye regions, but have symmetries thatresult in visually pleasing results. In FIG. 15A, the glint regions havebeen added somewhat below and to the right in eyes of a subject whosegaze is somewhat below horizontal and to the right. In FIG. 15B, glintregions have been added at the bottom of both of the iris regions.

In certain embodiments, rather than searching for eye centers withindigital image data, groups of similar pixels (e.g., color, intensity,location and/or size) are searched for within the eye regions of a facedetected within an acquired digital image. In certain embodiments, aneye center may be determined by searching for pixels around radialsymmetry maxima. IN this context, the following papers are incorporatedby reference:

R. Valenti, T. Gevers, Accurate eye center location through invariantisocentric patterns, IEEE Trans. PAMI 34 (9) (2012) 1785-1798.—usescircular isophote (i.e. curves formed by pixels with the samegrey-level) to vote for the eye center. The idea is that for a givenradially-symmetrical object, the isophotes are concentric one toanother, and the assumed center is the same, such that one may add thecontribution of each isophote and get the center of the circular object;and

L. Bai, L. Shen, and Y. Wang. Novel eye location algorithm based onradial symmetry transform. In ICPR, pages 511-514, 2006. In thistechnique, given the eye region, one finds the points with maximalradial symmetry. The radial symmetry is measured with aparticularization of Reisfeld's generalized symmetry transform (eq. (8)in the paper). So, for each point in the eye region, one measures theradial symmetry and the center is chosen to be the one associated withmaximum valuel; and

F. Timm and E. Barth. Accurate eye centre localisation by means ofgradients. In VISAPP, 2011. pp. 125-130. In this technique, one firstcomputes the edge the eye iris. Then, given a circular curve, one mayfind the center of the curve by accumulating votes from multiple chordpoints. A chord, in this embodiment, may be defined as a line orthogonalon the tangent of the chord in each point, from the given.

In certain embodiments, a radial symmetry discovered in one eye regionmay be synchronized within two eye regions or between the two regions ofthe facial region. The procedure of locating one eye center may besymmetrical while the other is added with synchronicity between the eyeregions an included calculation. Also, multiple eye regions may besynchronized for two or more faces in a digital image. An accuracymeasure may use a max error from multiple eyes, e.g., from left andright eyes.

In certain embodiments, the eye center need not be located precisely.For example, face symmetry (and thus similarity) may be used for facefeature point's and localization methods (e.g., ASM and the subsequentderivations) or for normalization methods that are used forpre-processing in face recognition. In those cases, either theinformation may be missing and/or may be filled by symmetry, or symmetrymay be used to minimize average localization error. In certainembodiments, localization of pixels that are similar and/or that havesome anatomical meaning, is performed advantageously to providebeautiful naturally looking eyes with glint regions.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited above and below herein, as well as thebackground, invention summary, abstract and brief description of thedrawings, are all incorporated by reference into the detaileddescription of the preferred embodiments as disclosing alternativeembodiments.

The following are incorporated by reference: U.S. Pat. Nos. 7,403,643,7,352,394, 6,407,777, 7,269,292, 7,308,156, 7,315,631, 7,336,821,7,295,233, 6,571,003, 7,212,657, 7,039,222, 7,082,211, 7,184,578,7,187,788, 6,639,685, 6,628,842, 6,256,058, 5,579,063, 6,480,300,5,781,650, 7,362,368, 7,551,755, 7,692,696, 7,469,071 and 5,978,519; and

U.S. published application nos. 2005/0041121, 2007/0110305,2006/0204110, PCT/US2006/021393, 2005/0068452, 2006/0120599,2006/0098890, 2006/0140455, 2006/0285754, 2008/0031498, 2007/0147820,2007/0189748, 2008/0037840, 2007/0269108, 2007/0201724, 2002/0081003,2003/0198384, 2006/0276698, 2004/0080631, 2008/0106615, 2006/0077261,2007/0071347, 20060228040, 20060228039, 20060228038, 20060228037,20060153470, 20040170337, and 20030223622, 20090273685, 20080240555,20080232711, 20090263022, 20080013798, 20070296833, 20080219517,20080219518, 20080292193, 20080175481, 20080220750, 20080219581,20080112599, 20080317379, 20080205712, 20090080797, 20090196466,20090080713, 20090303343, 20090303342, 20090189998, 20090179998,20090189998, 20090189997, 20090190803, 20090179999; andU.S. patent application Nos. 60/829,127, 60/914,962, 61/019,370,61/023,855, 61/221,467, 61/221,425, 61/221,417, 12/748,418, 61/182,625,61/221,455, and 12/479,658.

What is claimed is:
 1. A method of enhancing an appearance of a facewithin a digital image, comprising using a processor in: acquiring adigital image of a scene including a face, including capturing the imageusing a lens and an image sensor of a processing device that includesthe processor, or receiving said image following capture by anotherdevice that includes a lens and an image sensor, or a combinationthereof; identifying one or two groups of pixels that each include apupil region of an eye region within the face in the digital image;identifying a border between said pupil region and an iris or a scleraof said each eye region using luminance information; adjusting thedigital image including adding one or more glint pixels at a pupil sideof the border between said iris and said pupil to generate an enhancedimage; and displaying, transmitting, communicating or digitally storingor otherwise outputting the enhanced image or a further processedversion, or combinations thereof.